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Abstract 



Background/Context: The mission of Teaching SMART, a science inquiry professional 
development program for elementary school teachers, is to encourage the performance and 
persistence of all students, particularly female and minority youth, in elementary science. 
Teaching SMART provides instruction, hands-on training, and long-term technical assistance 
and support for third through fifth grade teachers in a given school to increase their awareness 
and comfort level in using equitable, hands-on inquiry, and exploration-based approaches to 
teaching science, ultimately enhancing student achievement. The Teaching SMART intervention 
has been fully developed and implemented in school districts around the country. 

Purpose/Objective/Research Question/Focus of Study: The study was designed to establish 
the efficacy of Teaching SMART (Teaching Science, Mathematics and Relevant Technologies); 
a science professional development program for teachers with students in grades 3 through 5. 
Teaching SMART promotes scientific inquiry and emphasizes the importance of equity, 
empowerment, exploration, and fun in the classroom. The paper focuses on two primary 
outcomes for the teachers and students who participated in the three year implementation of 
Teaching SMART: (1) the impact on science teaching practices of teachers and (2) effects on 
student achievement in science. 

Setting: There are 35 elementary schools ranging in size from 400 to 1 100 students in a large 
Florida school district, characterized by low-SES, diverse minority groups, and linguistic 
minority groups with three schools having a majority student population that is non-white. There 
are 18 Title I schools in the district with a free/reduced lunch eligibility rate greater than 50%. 
The mobility rate among students is 44%. In the fall 2005, there were approximately 62,768 
students attending pre-kindergarten through 12th grades, including 28,831 students at the 
elementary level in grades k-5. The racial composition of the elementary students is 78% White, 
non-Hispanic, 4% African American, 12% Hispanic, 6% other (including American Indian, 
Asian, and Multi-racial). 

The school district has three guiding principles, continuous progress, continuity of caring, and 
ensuring equity and excellence. The intent of continuous progress is to enhance students’ 
academic, social/emotional, and physical performance with a curriculum that allows students to 
progress at their own rate. The “blurring” of ages across grades results in multiage or non-graded 
classrooms. About 60 percent of the teachers in the study taught classes with multiple 
combinations of grade levels, including three and four, four and five, and three through five. The 
goal of continuity of caring allowed students to have individual teachers or teams of teachers 
across multiple years. Most of our schools’ buildings (14 of 20 schools) were designed with four 
classrooms having a common core area that included common restrooms, open area, and storage 
or support staff rooms called pods. The four teachers in a pod work together to varying degrees 
as a team. Some schools follow the continuous progress plan while others allowed teams to move 
students among the four teachers. In some teams we find teachers teaching by grade levels for 
science, while in others teachers taught multigrade science. 

When we implemented the program in the fall of 2005 the state testing of fifth grade science was 
included in each school’s accountability measure. The district also began a district- wide 
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professional development program based on Learning Focused Strategies (LFS), developed by 
Max Thompson and Julia Thompson. This program was implemented in groups of elementary 
schools each year for three years. The program focused on generic teaching strategies such as, 
essential questions, activating, teaching, and summarizing strategies. 

There was growing pressure placed upon teachers to focus on reading. Some principals would 
not allow teachers to use any science reading activities as part of their required uninterrupted 90 
minute reading/language arts block. The district uses an integrated thematic program. The 
themes for elementary intermediate science were clustered across three years by semester among 
science, social studies, and health, typically a science, social studies, and health related theme. 
The district curriculum map for the science included 60 days of science themes for year one, 80 
days for year two, and 55 days for year three. The district encouraged integration of content so 
teachers could include applicable science lessons as part of their social studies or health themes. 

Population/Participants/Subjects: This is an analysis of the first cohort of third grade students 
participating in three years of implementation with 129 initial teachers. In all, there are 964 third 
grade students from ten treatment and ten control schools. 

Intervention/Program/Practice: Teaching SMART is a comprehensive, three-year teacher 
professional development program designed to produce systemic changes in the classroom and 
the school through improving science education at the elementary school level. The professional 
development program is intended to change teaching practice in accord with the National 
Research Council’s National Science Education Standards. The program is designed to support 
the development of teachers’ confidence in teaching science through the development of their 
science content knowledge and pedagogical skills. In turn, students’ attitudes toward and 
confidence in science are expected to improve. Teacher professional development consists of 
three annual trainings with classroom follow-up visits by the resource teachers and half-day 
networking sessions. The program is delivered by district resource teachers who have been 
trained and monitored by Teaching SMART program staff. The Teaching SMART program 
provided teachers with 103 lessons that were adapted to follow the Teaching SMART format. 
The lessons were reorganized by the Teaching SMART resource teachers to align with the 
district’s three year thematic cluster program. There were an average of 45 Teaching SMART 
lessons recommend a year and 85 unduplicated recommended lessons across the three year 
cluster. At the conclusion of each level of teacher training, teachers signed a “contracf ’ 
stipulating participation in Teaching SMART activities that included completing minimum of 
two lessons per month. The resource teachers prepared kits for each teacher and school. These 
kits contained the majority of the materials needed to conduct the recommended activities. The 
resource teachers also would assist teachers by providing perishable supplies. These kits were 
“refurbished” each year by the resource teachers. 

Research design: A randomized controlled trial is used in which schools were randomly 
assigned to either a treatment or control condition. Teachers in treatment schools participate in 
the Teaching SMART professional development program while teachers in control schools 
participated in a business-as-usual condition. 
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Data collection and analysis: Baseline data were collected from teachers and their students in 
the fall of 2005 and then annually each spring. The items used in the teacher and student survey 
were adapted from the work of the Council of Chief State School Officers (Blank, Porter & 
Smithson, 2001). Scores from the instructional practice items were explored through factor 
analysis where five categories were created: empowering, hands-on, exploring, inquiry, and 
conventional practices. In this analysis, we focus on the reported impact of professional 
development on science inquiry and hands-on activities. Three measures of science achievement 
are used to evaluate the impact of the Teaching SMART professional development program on 
student outcomes. Two measures were developed by the Partnership for the Assessment of 
Standards-based Science (PASS) and consist of a performance task and a multiple choice 
assessment. Students’ responses to multiple-choice PASS items are computer scored while 
performance items are scored using analytical rubrics with high interrater reliability (Klein, 
Stecher, Shavelson, McCaffery, Ormseth, Bell, Comfort & Othman, 1998). The third measure is 
the state of Florida standardized test, the Florida Comprehension Assessment Test (FCAT). A 
repeated measures analysis of the Florida Comprehensive Achievement Test (FCAT) scores for 
treatment and control schools for 2005, 2006, and 2007 using HLM 6.06. 

Findings: 

Teachers’ instructional practice scores for Inquiry and Hands-On activities on the teacher survey 
that was administered at four time points (baseline fall 2005, spring 2006, spring 2007, and 
spring 2008) were analyzed (see Table 1). Overall, treatment teachers scored higher on inquiry 
[F = (3, 887) = 26.93, p<.005. Partial Eta Squared=.08] and hands-on activities [F = (3, 887) = 
18.41, p<.005. Partial Eta Squared=.06] (see Eigure 1 and 2) instructional practices than control 
teachers. The effect size of these measures is small explaining less than 10 percent of the 
variance. 

A repeated measures analysis of the Elorida Comprehensive Achievement Test (EC AT) fifth 
grade science scores for treatment (n=10) and control (n=9) schools for 2005, 2006, and 2007 
using HLM 6.06 indicated that the average initial score for schools was 290.2 (p < 0.001) and the 
average change for schools was 0.28 (p < 0.003) per unit time (year). The predicted difference in 
ECAT scores (Eigure 3) between treatment and control schools per unit time was 0.47 (greater if 
a treatment school), but this difference was not significant (p < 0.89). 

Conclusions: 

We are encouraged by the teachers reported change in practice and the concurrent student 
reports. The lack of significant effects on measures of student achievement may be due to a 
variety of factors including contextual pressures emphasizing reading and mathematics 
achievement. Repeated measures analyses of the recent ECAT scores at the student level may 
provide greater power. 

During the implementation of the Teaching SMART program there are a number of issues that 
may have influenced teacher practices and student achievement. Issues include the district 
context, and the Teaching SMART program structure and implementation. The school district’s 
support was initially very strong, but waned as the program progressed. The supervisor for 
elementary science was routinely present for the opening of each of the multiple beginning 
training sessions and on half of those trainings her supervisor spoke of the district’s support for 
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the program. These two initial people were promoted at the end of the first year. Their 
replacements, although very supportive, underwent a period of learning the details of the 
program. The district simultaneously implemented their district-wide Learning Focused 
Strategies (LFS) teacher professional development program. The Learning Focused Strategies is 
described as a research based school improvement framework of best practices that emphasizes 
planning, prioritized curriculum, instructional strategies, assessment and school organization to 
maximize student achievement. During the three years of Teaching SMART, successive schools 
were also participating in the district’s LFS program. From our interviews, observations of the 
Teaching SMART training, and feedback from the resource teachers we discern that teachers and 
principals sometimes viewed the Teaching SMART program as an additional burden. Other staff 
changes may have influenced teacher perceptions of the program. During the second year of the 
program 35 percent of the principals of participating schools were changed and during the third 
year of the program another 25 percent changed. Another major change during the second year 
was the loss of one of the three resource teachers due to a promotion to an assistant principal. 
Although these changes were handled smoothly, they tended to undermine the continuity of 
delivery and support. 

The district’s thematic curriculum structure for science and the increasing focus on reading and 
reading test scores are also an area of concern. The district’s thematic approach combined with 
Teaching SMART’S minimum requirement of two lessons per month may have provided an 
insufficient dosage of student participation in science. A recurrent theme of teacher comments 
was insufficient time for teaching science. 

The structure of the Teaching SMART resource teacher’s classroom visits may have contributed 
to some teachers’ apparent decrease in participation in classroom lessons. As part of the 
Teaching SMART program, the resource teachers decreased the number of classroom visits as 
well as the type of support they provided each year to encourage teachers to take on the 
responsibility of improving their science instruction so they could be relatively independent of 
outside support by the end of the program. Many teachers progressed along with the program’ s 
expectations, but a number of teachers appeared to need additional support to continue their 
growth in science instruction. 

Delivery of the professional development by the resource teachers was always well planned and 
structured based upon the experience of the Teaching SMART program director. We found the 
implementation among repeated trainings was uniform, but that problems the resource teachers 
had were not always adequately addressed. Initially the resource teachers tended to deliver the 
training in didactic manner undermining the inquiry base of the program. 

Overall these context and programmatic issues may have slowed the growth of teachers 
participating in the program. This slowed growth for teachers combined with a lag in increases in 
student achievement scores offers some optimism for the success of the Teaching SMART 
program. 
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Tables and Figures 



Table 1. Comparison of treatment and control teachers’ hands-on activities and inquiry scores 



Factor 


Wave 




Treatment 




Control 






M 


N 


SD 


M 


N 


SD 


ES 


Description 


Hands-On 


Fall 2005 


2.88 


173 


0.68 


2.8 


78 


0.56 


0.12 


negligible 




Spring 2006 


3.18 


168 


0.54 


2.92 


101 


0.54 


0.48 


medium 




Spring 2007 


3.26 


109 


0.43 


3.08 


73 


0.51 


0.39 


small 




Spring 2008 


3.29 


58 


0.42 


3.07 


47 


0.46 


0.50 


medium 


Treatment 


T F05 to S06 


2.88 


173 


0.68 


3.18 


168 


0.54 


0.49 


medium 




T F05 to S07 


2.88 


173 


0.68 


3.26 


109 


0.43 


0.64 


medium 




T F05 t SOS 


2.88 


173 


0.68 


3.29 


58 


0.42 


0.65 


medium 


Control 


C F05 to S06 


2.8 


78 


0.56 


2.92 


101 


0.54 


0.22 


small 




C F05 to S07 


2.8 


78 


0.56 


3.08 


73 


0.51 


0.52 


medium 




C F05 t SOS 


2.8 


78 


0.56 


3.07 


47 


0.46 


0.51 


medium 


Inquiry 


Fall 2005 


2.68 


173 


0.62 


2.71 


78 


0.53 


0.05 


negligible 




Spring 2006 


2.87 


168 


0.58 


2.73 


101 


0.5 


0.25 


small 




Spring 2007 


3.04 


109 


0.48 


2.84 


73 


0.53 


0.40 


small 




Spring 2008 


3.17 


58 


0.58 


2.82 


47 


0.48 


0.65 


medium 


Treatment 


T F05 to S06 


2.68 


173 


0.62 


2.87 


168 


0.58 


0.32 


small 




T F05 to S07 


2.68 


173 


0.62 


3.04 


109 


0.48 


0.63 


medium 




T F05 t SOS 


2.68 


173 


0.62 


3.17 


58 


0.58 


0.80 


large 


Control 


C F05 to S06 


2.71 


78 


0.53 


2.73 


101 


0.5 


0.04 


negligible 




C F05 to S07 


2.71 


78 


0.53 


2.84 


73 


0.53 


0.24 


small 




C F05 t SOS 


2.71 


78 


0.53 


2.82 


47 


0.48 


0.21 


small 
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Hands-On Means 



Figure 1. Teacher means for hands-on activity scores 




Condition 

^ Control 
^Treatment 
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Inquiry Means 



Figure 2. Teacher means for inquiry activity scores 




Condition 

^ Control 
^Treatment 
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Figure 3. Comparison of treatment and control student FCAT school scores 




TREAT1CT = 0 
TREAT1CT = 1 
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